Content aligned block-based deduplication

ABSTRACT

A content alignment system according to certain embodiments aligns a sliding window at the beginning of a data segment. The content alignment system performs a block alignment function on the data within the sliding window. A deduplication block is established if the output of the block alignment function meets a predetermined criteria. At least part of a gap is established if the output of the block alignment function does not meet the predetermined criteria. The predetermined criteria is changed if a threshold number of outputs fail to meet the predetermined criteria.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. §119(e)of U.S. Provisional Patent Application No. 61/388,531, filed on Sep. 30,2010, and entitled “CONTENT ALIGNED BLOCK-BASED DEDUPLICATION,” thedisclosure of which is hereby incorporated by reference in its entirety.

BACKGROUND

Computers have become an integral part of business operations such thatmany banks, insurance companies, brokerage firms, financial serviceproviders, and a variety of other businesses rely on computer networksto store, manipulate, and display information that is constantly subjectto change. Oftentimes, the success or failure of an importanttransaction may turn on the availability of information that is bothaccurate and current. Accordingly, businesses worldwide recognize thecommercial value of their data and seek reliable, cost-effective ways toprotect the information stored on their computer networks.

In corporate environments, protecting information is generally part of aroutine process that is performed for many computer systems within anorganization. For example, a company might back up critical computingsystems related to e-commerce such as databases, file servers, webservers, and so on as part of a daily, weekly, or monthly maintenanceschedule. The company may also protect computing systems used by each ofits employees, such as those used by an accounting department, marketingdepartment, engineering department, and so forth.

As such, enterprises are generating ever increasing volumes of data andcorresponding storage requirements. Moreover, enterprise storage systemsare typically distributed over one or more networks, such as wherebackup storage is remote from client computers. In such situations,storage system activity can place heavy demands on available networkbandwidth.

SUMMARY

In response to these challenges, one technique developed by storagesystem providers is data deduplication. Deduplication typically involveseliminating or reducing the amount of redundant data stored andcommunicated within a storage system, improving storage utilization. Forexample, a data segment can be divided into units of a chosengranularity (e.g., files or data blocks). As new data segments enter thesystem, the data units can be checked to see if they already exist inthe storage system. If the data unit already exists, instead of storingand/or communicating a duplicate copy, the storage system stores and/orcommunicates a reference to the existing data unit. Thus, deduplicationcan improve storage utilization, system traffic (e.g., over a networkedstorage system), or both.

One method of dividing the data segment into units is to use fixed-sizeblocks at fixed-intervals. As an example, a 1 MB data segment can besubdivided into eight 128 kB blocks. Each of the 128 kB blocks iscompared to each other as well as other 128 kB blocks within the storagesystem in order to identify all the identical blocks. However, if a datablock differs from previously stored data blocks by just one byte, theresult may be no matching data blocks are found, and no data isdeduplicated, even though all the data is identical except the one byte.

In view of the foregoing a need exists to more efficiently determine theappropriate alignment for each block of data, thereby increasing theamount of data that is deduplicated and decreasing storage requirements.According to certain aspects, a sliding window alignment function isperformed on data segments to establish block alignments based on thecontent of the data. The established data blocks can be fixed length,for example. In some embodiments, gaps of data not belonging to anydeduplicated blocks exist between the deduplicated data blocks.

In determining the appropriate alignment some type of predeterminedcriteria is generally used. Using different criteria in somecircumstances can dramatically affect the number of blocks that arededuplicated. Thus, according to certain embodiments, the predeterminedcriteria is dynamically refined during the deduplication process,providing improved storage utilization.

In certain embodiments, a method is disclosed for defining deduplicationblock alignments within a data segment. The method includes iterativelyperforming a deduplication block alignment function on data within asliding window in a data segment. For each iterative performance of thededuplication block alignment function, the method further includes inresponse to determining that the output of the deduplication blockalignment function performed on the data within the sliding windowsatisfies a predetermined criteria establishing with one or morecomputer processors a deduplication data block having a predeterminedblock size. For each iterative performance of the deduplication blockalignment function, the method further includes in response todetermining that the output of the deduplication block alignmentfunction performed on the data within the sliding window does notsatisfy the predetermined criteria defining at least a portion of a gapof data not belonging to a deduplication data block.

In certain embodiments, a system is disclosed for defining deduplicationblock alignments within a data segment. The system comprises adeduplication block alignment module executing in one or more processorsand configured to iteratively perform a deduplication block alignmentfunction on data within a sliding window in a data segment. The blockalignment module is further configured for each iterative performance ofthe block alignment function to establish a deduplication block having apredetermined block size in response to determining that the output ofthe deduplication block alignment function performed on the data withinthe sliding window satisfies a predetermined criteria. The blockalignment module is further configured for each iterative performance ofthe block alignment function to define at least a portion of data notbelonging to a deduplication block in response to determining that theoutput of the block alignment function performed on the data within thesliding window does not satisfy the predetermined criteria, for eachiterative performance of the block alignment function.

In certain embodiments, a method is disclosed for defining deduplicationblock alignments within a data segment. The method includes iterativelyperforming a block alignment function on data within a sliding window ina data segment. For each iterative performance of the alignmentfunction, the method further includes establishing with one or morecomputer processors a deduplication data block having a predeterminedblock size and moving the sliding window relative to the data segment byan amount based on the predetermined block size before performing thenext iteration in response to determining that the output of the blockalignment function performed on a current window of data satisfies apredetermined criteria. For each iterative performance of the alignmentfunction, the method further includes in response to determining thatthe output of the block alignment function performed on the currentwindow of data does not satisfy the predetermined criteria moving thesliding window relative to the data segment by an incremental amountbefore performing the next iteration and without establishing adeduplication data block, wherein gaps of data not belonging to anydeduplication data block exist between established deduplication datablocks following performance of the block alignment function across thedata segment.

In certain embodiments, a method is disclosed for determiningdeduplication block alignments within a data segment. The methodincludes selecting a first range of possible output values of adeduplication block alignment function which indicate that a blockalignment has been found and iteratively performing the deduplicationblock alignment function on data within a sliding window in a datasegment. For each iterative performance of the alignment function, themethod further includes determining with one or more computer processorswhether the output of the deduplication block alignment functionperformed on the data within the sliding window falls within the firstrange. For each iterative performance of the alignment function, themethod further includes establishing with one or more computerprocessors a deduplication data block having a predetermined block sizein response to determining that the output of the block alignmentfunction falls within the first range. The method further includesselecting a second range of output values for the block alignmentfunction which indicate that a block alignment has been found, theselection of the second range performed in response to determining, fora threshold number of iterations, that the output of the block alignmentdoes not fall within the first range, wherein the second range is usedinstead of the first range for subsequent iterations of the blockalignment function.

In certain embodiments, a deduplication system is disclosed forconfigured determining deduplication block alignments within a datasegment. The system includes a block alignment module executing in oneor more processors. The block alignment module is configured to select afirst range of output values for a deduplication block alignmentfunction which indicate that a block alignment has been found, anditeratively perform the deduplication block alignment function on datawithin a sliding window in a data segment. The block alignment module isfurther configured for each iterative performance of the block alignmentfunction to determine whether the output of the deduplication blockalignment function performed on data within the sliding window fallswithin the first range, and establish a deduplication data block with apredetermined block size in response to determining that the output ofthe deduplication block alignment function performed on the data withinthe sliding window falls within the first range. The system furtherincludes a criteria adjustment module configured to select a secondrange of output values for the block alignment function which indicatethat a block alignment has been found, the selection of the second rangein response to the block alignment module determining, for a thresholdnumber of iterations, that the output of the block alignment performedon the data within the sliding window does fall within the first range,wherein the second range is used instead of the first range forsubsequent iterations of the block alignment function.

In certain embodiments, a method is disclosed for refining criteria fordetermining deduplication block alignments within a data segment. Themethod includes selecting a first range of output values for adeduplication block alignment function which indicate that a blockalignment has been found, and iteratively performing a block alignmentfunction on data within a sliding window in a data segment. For eachiterative performance of the alignment function, the method furtherincludes in response to determining with one or more computer processorswhether the output of the block alignment function performed on acurrent window of data falls within the first range establishing adeduplication data block having a predetermined block size, and movingthe sliding window in a first direction relative to the data segment byan amount based on the predetermined block size before performing thenext iteration. For each iterative performance of the alignmentfunction, the method further includes in response to determining thatthe output of the block alignment function performed on the currentwindow of data does not fall within the first range for a thresholdnumber of iterations, selecting a second range of output values for theblock alignment function which indicate that a block alignment has beenfound, and moving the sliding window over data relative to the datasegment in a second direction opposite the first direction beforeperforming the next iteration using the second range, wherein the nextiteration is performed on data on which the block alignment function waspreviously performed using the first range.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates components of an examplestorage system configured to implement techniques compatible withembodiments described herein.

FIG. 2 is a block diagram illustrative of an example data segment duringdeduplication in accordance with embodiments described herein.

FIGS. 3 and 4 are block diagrams illustrative of other examples datasegments during deduplication in accordance with embodiments describedherein.

FIG. 5 is a block diagram showing an expanded view of an example datasegment including block alignments created in accordance withembodiments described herein.

FIG. 6 is a flow diagram illustrative of one embodiment of a routineimplemented by a storage system for defining deduplication blockalignments within a data segment.

FIG. 7 is a flow diagram illustrative of one embodiment of a routineimplemented by a storage system for deduplicating data blocks within adata segment.

FIG. 8 is a flow diagram illustrative of another embodiment of a routineimplemented by a storage system for defining deduplication blockalignments within a data segment.

FIG. 9 is a flow diagram illustrative of another embodiment of a routineimplemented by a storage system for defining deduplication blockalignments within a data segment.

FIG. 10 is a flow diagram illustrative of another embodiment of aroutine implemented by a storage system for defining deduplication blockalignments within a data segment.

DETAILED DESCRIPTION

Generally described, the present disclosure is directed to a system,method, and computer-readable non-transitory storage medium for definingdeduplication block alignments within a data segment and deduplicatingdata blocks within a data segment. Specifically, aspects of thedisclosure will be described with regard to iteratively determiningblock alignments for a data segment that is to be deduplicated. Althoughvarious aspects of the disclosure will be described with regard toexamples and embodiments, one skilled in the art will appreciate thatthe disclosed embodiments and examples should not be construed aslimiting.

As described above, determining the alignment of deduplication blocksbased on the content of the data according to techniques describedherein can significantly increase the probability that the data will bededuplicated, and that a data segment will use less storage. The presentdisclosure describes certain embodiments that determine the alignment ofdeduplication blocks using a block alignment function. A sliding windowmoves across the data segment and the block alignment function isperformed on the data in the sliding window. Deduplication blocks areformed when the output of the block alignment function meetspredetermined criteria, which may also be referred to as a “hit.” Gaps,or parts of gaps, are formed when the output of the block alignmentfunction does not meet predetermined criteria i.e. when there is no“hit.”

After the sliding window has moved across the data segment, the datasegment includes deduplication blocks and gaps between some or all ofthe deduplication blocks. The deduplication blocks can be a fixed-sizeor variable-size. The gaps between the deduplication blocks will likelyvary in size. The size of the gaps are reflective of the number ofiterations between the block alignment function returning hits. Anexample, this will be described in greater detail below with referenceto FIG. 5.

The deduplication blocks are then compared with each other and otherpreviously stored deduplication blocks to determine if they containidentical data. The gaps, however, may not under go this deduplicationprocess. After the deduplication blocks are compared, the deduplicationblocks with identical data as other deduplication blocks are replacedwith references, often pointers, to the other deduplication blocks. Theremaining data segment includes the gaps, references to thededuplication blocks with identical data, and the deduplication blocksthat did not have identical data stored elsewhere. Such techniquesgenerally provide improved storage utilization in an efficient manner.

By dynamically aligning the deduplication blocks, the storage systemwill increase the likelihood of finding identical data, which can thenbe replaced with a reference to the other stored data. Thus, the amountof memory required to store the data segment can be significantlyreduced, leading to significant cost-savings for enterprises.

In addition, the description includes embodiments for altering, ortuning, the predetermined criteria during the block alignment phasebased on the data. As the sliding window moves across the data, thestorage system can determine that there are insufficient hits using thecurrent block alignment function. The storage system can modify thepredetermined criteria to increase the likelihood of hits and/or modifythe block alignment function. Thus, the storage system can dynamicallyadapt to the data segment, further leading to an increased probabilityof finding identical data and reducing the amount of memory used by thedata segment. In the present disclosure, a data segment can generallyrefer to a set of data that includes smaller components. A data segmentmay be of any size, and may be in the form of stored data or streamingdata. A data segment may take the form of parts of a file, an entirefile, or multiple files.

In the present disclosure, a sliding window can generally refer to aboundary used to analyze different parts of a data segment.

In the present disclosure, a block alignment function can generallyrefer to any type of function that can be used to determine blockalignments. A specific example of a block alignment function is a hashfunction, particularly weak hash functions.

In the present disclosure, a block comparison function can generallyrefer to any type of function capable of comparing two different sets ofdata within a data segment. A block comparison function may be as simpleas comparing each bit, or byte, of data from one set of data to eachbit, or byte, of data form another set of data. A more complex type ofblock comparison function involves comparing hash function values ofblocks, particularly those of strong hash functions.

In the present disclosure, deduplication can generally refer toidentifying and removing duplicative data within a data segment.Deduplication may refer to the identifying step, the removing step, orboth.

Illustrative explanations of several terms used throughout thedisclosure are provided above. While these meanings apply to therespective terms as used with respect to certain embodiments, it will beappreciated that the meanings can vary depending on the embodiment.Additionally, the meanings of these and other terms used herein will beunderstood in view of their usage throughout the entirety of thedisclosure.

FIG. 1 illustrates a block diagram of an example network storagearchitecture compatible with embodiments described herein. The system100 is configured to perform storage operations on electronic data in acomputer network. As shown, the storage system 100 includes a storagemanager 102 and one or more of the following: a client 185, aninformation store 190, a data agent 195, a media agent 105, an indexcache 110, and a storage device 115.

A data agent 195 can be a software module that is generally responsiblefor archiving, migrating, and recovering data of a client computer 185stored in an information store 190 or other memory location. Each clientcomputer 185 has at least one data agent 195 and the storage system 100can support many client computers 185. The storage system 100 provides aplurality of data agents 195 each of which is intended to backup,migrate, and recover data associated with a different application. Forexample, different individual data agents 195 may be designed to handleMicrosoft Exchange™ data, Microsoft Windows file system data, and othertypes of data known in the art. If a client computer 185 has two or moretypes of data, one data agent 195 may be implemented for each data typeto archive, migrate, and restore the client computer 185 data.

The storage manager 102 is generally a software module or applicationthat coordinates and controls the system. The storage manager 102communicates with all elements of the storage system 100 includingclient computers 185, data agents 195, media agents 105, and storagedevices 115, to initiate and manage system backups, migrations,recoveries, and the like.

A media agent 105 is generally a software module that conducts data, asdirected by the storage manager 102, between locations in the storagesystem 100. For example, the media agent may conduct data between theclient computer 185 and one or more storage devices 115, between two ormore storage devices 115, etc. The storage devices 115 can include atape library, a magnetic media storage device, an optical media storagedevice, or other storage device. Although not shown in FIG. 1, one ormore of the media agents 105 may also be communicatively coupled to oneanother.

Each of the media agents 105 can be communicatively coupled with andcontrol at least one of the storage devices 115. The media agent 105generally communicates with the storage device 115 via a local bus. Insome embodiments, the storage device 115 is communicatively coupled tothe media agent(s) 105 via a Storage Area Network (“SAN”).

Further embodiments of storage systems such as the one shown in FIG. 1are described in application Ser. No. 10/818,749, now U.S. Pat. No.7,246,207, issued Jul. 17, 2007, which is incorporated by referenceherein. In various embodiments, components of the storage system 100 maybe distributed amongst multiple computers, or one or more of thecomponents may reside and execute on the same computer.

Furthermore, components of the storage system 100 of FIG. 1 can alsocommunicate with each other via a computer network. For example, thenetwork may comprise a public network such as the Internet, virtualprivate network (VPN), token ring or TCP/IP based network, wide areanetwork (WAN), local area network (LAN), an intranet network,point-to-point link, a wireless network, cellular network, wireless datatransmission system, two-way cable system, interactive kiosk network,satellite network, broadband network, baseband network, combinations ofthe same or the like.

Additionally, the various components of storage system 100 may beconfigured for deduplication. For example, one or more of the clients185 can include a deduplicated database (DDB). The data stored in thestorage devices 115 may also be deduplicated. For example, one or moreof the media agents 105 associated with the respective storage devices115 can manage the deduplication of data in the storage devices 115.

FIG. 2 is a block diagram illustrative of a data segment 200 duringdeduplication. The deduplication techniques described with respect toFIG. 2 may be performed by any number of different components of storagesystem 100, described above with reference to FIG. 1, or some otherstorage system. For example, deduplication may be performed by theclient 185, data agent 195, storage manager 102 or any one of the mediaagents 105. Furthermore, the deduplication may be performed on streamingdata being sent to or from the client 185, storage manager 102, and/ormedia agent 105. Deduplication can also be performed on data residing inany of the client 185, storage manager 102, media agents 105, and/orstorage devices 115. In one example scenario, one or more of the storagedevices 115 comprise backup storage, and the media agents 105 generallymanage deduplication of backup data that is stored on the storagedevices 115. Generally, any of the block alignment techniques describedherein such as those described with respect to any of FIGS. 2-10 may beperformed by a deduplication block alignment module executing on one ormore of the components in the system. For example, a block alignmentmodule may be implemented on the storage manager, one or more of themedia agents, or a combination thereof.

As shown in FIG. 2, a data segment 200 may be made up of any number ofdata subcomponents 202. For ease of reference, each data subcomponent202 in the example of FIG. 2 represents one byte of information,however, the data subcomponents 202 can take any number of differentforms such as bits, bytes or multiples thereof. As discussed previouslythe data segment 200 may be streaming data or stored in computer memory.The data segment may comprise a file, multiple files or a portion of afile.

As part of deduplication, a block alignment function is performed on oneor more data subcomponents 202 within the data segment 200. A slidingwindow can be used to determine what data is to be used. The slidingwindow may be any number of sizes. For example, in various embodiments,the sliding window can be 32 kilobytes (kB), 64 kB, 128 kB, or somesmaller or larger value. In the example of FIG. 2, the sliding windowcovers six subcomponents of data, or six bytes. Thus, there are sixbytes of data within the sliding window at any given time. However, itis to be understood that the sliding window may be any size, and theexample illustrated in FIG. 2 is not to be construed as limiting. Thesliding window may also vary in size as it moves relative to the datasegment.

The block alignment function used can be any number of differentfunctions to determine an appropriate alignment for a deduplicationblock. In the example of FIG. 2, a hash function is used. In otherembodiments, the block alignment function may comprise or output amessage digest, checksum, digital fingerprint, digital signature orother sequence of bytes that substantially uniquely identifies therelevant data block. When a hash function is used, the hash functionreturns a value for the data within the sliding window. As illustratedin FIG. 2, a first hash 204 is performed on the data at the beginning ofthe data segment 200, which is within the sliding window. The firsthash, or hash1, 204 returns a hash value. The hash value is thencompared with a predetermined range of values. If the hash value fallswithin the predetermined range a deduplication block is established., asillustrated at 218. While the example system described with respect toFIG. 2 compares the hash value to a predetermined range, the system mayevaluate the hash to determine whether it meets a variety ofpredetermined criteria, as will be discussed in greater detail herein.

The size of the deduplication blocks 222, 224, 226, 228 in theillustrated embodiment correspond with the size of the sliding window(e.g., six bytes). If the hash value does not fall within thepredetermined range then the sliding window moves relative to the datasegment by an incremental value. In this example, the incremental valueis representative of one data subcomponent 202, or one byte. However theincremental value can be larger or smaller than one subcomponent. Forexample, the incremental value could be one bit, byte, or multiplesthereof.

Once the sliding window moves relative to the data segment 200, anotherhash is performed on the data that is within the sliding window afterthe move. As illustrated in FIG. 2, the data in the second hash, orhash2, 206 is similar to the data in the first hash 204 except that thesecond byte of data in the first hash 204 is the first byte of data inthe second hash. In addition, the last byte of data in the first hash204 is the second-to-last byte of data in the second hash 206. In otherwords, in the example illustrated in FIG. 2 the sliding window movesrelative to the data segment to the right by one byte, which representsthe incremental value in this example. The second hash returns a hashvalue which is compared with the predetermined range. Once again, if thehash value falls within the predetermined range, a deduplication blockis established 218. If not, the sliding window slides by one incrementalvalue, i.e. one byte. The process is continued until a hash returns avalue that falls within the predetermined range, or until the slidingwindow has moved across the entire data segment 200. Returning a hashvalue within the predetermined range can also be referred to as a “hit”.A hit occurs in FIG. 2 with the third hash, or hash3, 208.

In the example illustrated in FIG. 2, the third hash 208 returns a hitwithin the predetermined range, and block1 222 is established based onthe hit. Block1 222 designates the data that returned the hit, and isthe same size as the sliding window. However, it is envisioned that thededuplication block size can vary from the sliding window size. In thisexample the data that returned the hit begins at the third byte and endsat the eighth byte. Thus, block1 222 is made up of six bytes andsimilarly begins at the third byte and ends at the eighth byte. Theexample illustrated in FIG. 2 shows that additional hits occurred at thehash5 212, hash(N−1) 214 and hashN 216. From those hits block2 224,block(N−1) 226, and blockN 228 are established, respectively.

With continued reference to FIG. 2, after the block alignment functionhas performed on the entire data segment 200, data segment 200 has atleast four deduplication blocks: block1 222, block2 224, block(N−1) 226,and blockN 228. In addition to the deduplication blocks, data segment200 contains gaps between some of the deduplication blocks. The gapsbetween deduplication blocks represent unsuccessful hashes, or hashesthat did not return a hit. For instance, gap 242 is made up of two bytesassociated with hash1 and hash2, which did not return hits. Gap 244 ismade up of one byte, associated with hash4, which also did not return ahit. Gap 246 represents leftover data that could not become part of adeduplication block after blockN 228 was established because there wasinsufficient data left to establish another deduplication block.

As discussed, in other embodiments the deduplication blocks may belarger or smaller than the sliding window. As one illustrative example,the deduplication system performs a hash function on a 64 kB slidingwindow of data and creates a 128 kB deduplication block. In someembodiments, one or more of the size of the deduplication data blocksand the size of the sliding window may be configurable, either manuallyby a user, or automatically by the deduplication system itself. Forexample, the size of the blocks or the size of the sliding window may beadjusted based on a measured deduplication ratio or other appropriateparameter.

As shown, the system may align the starting point of the deduplicationblocks 222, 224, 226, 228 at the beginning of the respective slidingwindow. For example, the deduplication block 222 begins at the thirdbyte of data, which corresponds to the first byte of the sliding windowof data on which the hash3 208 was performed. In other embodiments, thededuplication block alignment starting points can be placed at someother position relative to the sliding window. For example, the systemcan align the starting point of the deduplication blocks at some otherpoint within the sliding window (e.g., in the middle of the slidingwindow). In yet other instances, the beginning of the deduplicationblocks is defined at some point outside of the sliding window. Forexample, in one embodiment, the system establishes deduplication blockssuch that they begin at a position in the data segment 200 before therespective sliding window. The deduplication blocks in such anembodiment can extend from this starting alignment to include some orall of the data within the sliding window.

Moreover, the data blocks can extend beyond the sliding window. Forexample, in the above embodiment where the deduplication system performsa hash function on a 64 kB sliding window of data and creates a 128 kBdeduplication block, the deduplication block can extend beyond thesliding window, depending on the starting alignment. For instance, wheresuch a block is aligned at the beginning of the sliding window, theblock extends 64 kB beyond the sliding window.

Once the deduplication blocks for the data segment have beenestablished, the deduplication blocks are deduplicated. To deduplicatethe blocks a block comparison function can be performed on the differentdeduplication blocks. For example, the block comparison function cancompare each byte of data in each deduplication block. Alternatively,the block comparison function can be a hash function. Thus, a hashfunction, different from the hash function used to establish thededuplication blocks, can be used to compare the deduplication blocks.Similar to what is described above, the hash function for each blockreturns a hash value. The hash value of each deduplication block can becompared. If the hash function of two deduplication blocks returns thesame hash value, it is highly probable that the two deduplication blockscontain equivalent data. Thus, the data of one of the deduplicationblocks can be removed from the data segment and replaced with areference to the other deduplication block. In this way, the amount ofdata stored in the data segment can be reduced. In addition, the hashvalues of the deduplication blocks of the data segment can be comparedwith hash values of deduplication blocks from other data segments. Whilecomparing the hash values of the deduplication blocks of the datasegment with the hash values of deduplication blocks from other datasegments can increase complexity and require additional resources, itcan also increase the likelihood of finding equivalent hash values.

In the example illustrated in FIG. 2, an equivalent hash value is foundfor the hash value of block1 222, block2 224 and block(N−1) 226. Thus,the data of block1 222, block2 224 and block(N−1) 226 is replaced withreferences B1 234, B2 236, and BN−1 238. The equivalent hash value canbe the same or different for each of block1 222, block2 224 andblock(N−1) 226. In addition, no equivalent hash value was found forBlockN 228. Thus, the data of blockN 228 is not replaced with areference and blockN 228 remains as part of the deduplicated datasegment 232. Thus, in the example illustrated in FIG. 2, after thededuplication process, data segment 232 contains gap 242, reference B1234, gap 244, reference B2 236, reference BN−1 238, blockN 228 and gap246.

Although not illustrated in FIG. 2, deduplication blocks for the entiredata segment need not be created before the blocks are deduplicated. Forexample, once hash1 204 returns a hit, block1 222 can be created andimmediately deduplicated. In other words, prior to hash4 210, thestorage system can compare the hash value of block1 222 with the hashvalue of other deduplication blocks to determine if block1 222 can bereplaced with reference B1 234. In various embodiments, deduplicationoccurs after creating deduplication block alignments for multipleblocks, an entire segment, multiple segments, a file comprised ofmultiple segments, or some other granularity of data. Thus, the processfor establishing deduplication blocks and deduplicating data segmentscan occur in any number of ways without departing from the spirit andscope of the disclosure.

In another embodiment, the storage system moves the sliding windowrelative to the data segment, and performs the block alignment functionfor all potential deduplication blocks before determining if the outputof the block alignment function for the potential deduplication blocksmeets the predetermined criteria. In such an embodiment, the storagesystem can evaluate the different outputs to determine the criteria tobe used as the predetermined criteria. In this way, the storage systemcan potentially reduce the number of gaps within the deduplicationblocks.

Because the gaps 242, 244, 246 of data are not included in adeduplication block, the gaps 242, 244, 246 generally represent data inthe data segment that is not used in the deduplication process. However,the blocks are aligned according to the alignment function based on thecontent of the data, improving the odds that the deduplication blockswill be redundant. Thus, while all of the data may not be used in thededuplication process, the overall storage utilization is improvedaccording to embodiments described herein. Moreover, in certainembodiments, the deduplication blocks all have the same, fixed length.Using fixed length blocks can reduce the computational overhead involvedin creating and comparing deduplication blocks for redundancies and/orsimplify the management and storage of the deduplication blocks oncecreated, among providing other advantages.

FIG. 3 illustrates another example of a data segment 300 duringduplication. In the example illustrated in FIG. 3, the block alignmentshave already been determined i.e. the deduplication blocks and gaps havealready been established. In FIG. 3, after the block alignments havebeen established, the data segment 300 is contains gap 312, block1 304,gap 314, block2 306, gap 316 and blockN 308. Similar to FIG. 2, the gapsbetween deduplication blocks represent hashes that did not return valueswithin the predetermined range, or outputs of the block alignmentfunction that did not meet the predetermined criteria. Thus, betweenblock1 304 and block2 306 there were several unsuccessful hashesoccurred, which resulted in gap 314.

Also similar to FIG. 2, FIG. 3 shows the data segment 300 afterdeduplication, represented as deduplicated segment 320. The deduplicatedsegment 320 is contains gap 312, reference B1 322, gap 314, reference324, gap 316 and reference BN. As discussed earlier, the referencesrefer to other deduplication blocks that contain data equivalent to thedata in the deduplication blocks corresponding to the references. Thus,during deduplication, a second hash function of block1 304, block2 315and blockN 308 return a hash value equivalent to a hash value of anotherdeduplication block. Thus, block1 304, block2 315 and blockN 308 arereplaced with references to the deduplication block with the equivalenthash value. It should be noted that block1 304, block2 315 and blockN308 may each be equivalent to the same or different deduplication blocksand references B1 322, B2 324, and BN 326 may all refer to differentdeduplication blocks or the same deduplication block.

FIG. 4 illustrates another example of a data segment 400 after the blockalignments have already been determined. In the example illustrated inFIG. 4, the initial block alignment function resulted in block1 404,block2 406 and blockN 408. A large gap, similar to gap 314 of FIG. 3 isestablished. However, to reduce the number of gaps, a storage system candetermine that once a gap is as large as a deduplication block, adeduplication block should be established from the gap. Thisdetermination and establishment of an additional deduplication block isillustrated with blockX 410. In the example illustrated in FIG. 4, thestorage system determines that the gap between block1 404 and block2 406is large enough to allow the establishment of blockX 410, despite thatthere were no hits between block1 404 and block2 406. To accomplishthis, the storage system tracks the number of unsuccessful hits. Oncethe number of hits reaches some threshold, the storage system canestablish the deduplication block.

In the example illustrated in FIG. 4, the deduplicated data segment 420contains gap 412, reference B1 422, blockX 410, 414, reference B2 424,gap 416 and reference BN 426. Thus, in the example illustrated in FIG.4, during the block comparison function, there was not another blockthat returned a hash value equal to that of blockX 410. Similar to thededuplicated segments illustrated in FIG. 2 and FIG. 3, each of thereferences B1 422, B2 424 and BN 426 refer to deduplication blocks withdata equivalent to that found in block1 404, block2 406 and blockN 408,respectively. In the example illustrated in FIG. 4, the references B1422, B2 424 and BN 426 can refer to deduplication blocks in other datasegments or can point to blockX 410. In another scenario, the systemdoes find a hit for blockX when deduplicating the blocks 418, and thededuplicated segment 420 therefore includes a reference to blockXinstead of blockX itself.

FIG. 4 can also be used to illustrate another function that can beperformed on data segment 402. As the sliding window moves relative tothe data segment 400, a counter can be used to track the number of timesa hash does not return a hit. If a threshold number of hashes fails toreturn a hit the predetermined range or other criteria can be adjusted.In one embodiment, adjusting the predetermined criteria can includeexpanding the range of values considered to be a hit. In anotherembodiment, the block alignment function can be changed entirely. Inanother embodiment, the sliding window size can be adjusted. Todetermine how to expand the range of values, or otherwise change thepredetermined criteria, the hash values of previous hashes can beexamined. Thus, in the example illustrated in FIG. 4, block1 404 returnsa hit, but the following several hashes fail to return a hit. Thestorage system analyzes the hash values of the previous unsuccessfulhashes and expands, or otherwise adjusts the range to increase the hits.In the example illustrated in FIG. 4, a threshold of six no-hits isused. In other words, after six iterations without returning a hit, thestorage system can adjust the range of values considered to be a hit. Inthe example illustrated in FIG. 4, after block1 404, six hashes areperformed, all of which fail to return a hit. The storage systemanalyzes the values of the hits and adjusts the range so that at leastone additional hit occurs. Adjusting the range results in blockX 410being established. After adjusting the range of values, the slidingwindow continues. In one embodiment, after adjusting the predeterminedcriteria, the storage system can move the sliding window relative to thedata segment to the beginning of the data segment to re-analyze thehashes in light of the changed criteria. In another embodiment, afterthe adjusting the predetermined criteria, the storage system canreevaluate the hash values of the previous hashes and establish any newdeduplication blocks in light of the reevaluation before continuing.Additionally, in embodiments where the range of hash values is expanded,the modified range includes the values that were in the original rangeand there is thus a relatively high probability of finding a matchingsuitable block boundary.

Continuing on with the example of FIG. 4, at least two additional hitsoccur with block2 406 and blockN 408. In another embodiment, once thepredetermined criteria is adjusted, the sliding window can return to thebeginning of the data segment 400. In an embodiment where the blockalignment function is adjusted, the sliding window can begin at thebeginning of the data segment. In an alternative embodiment, once thepredetermined criteria is adjusted, only some of the data isreevaluated. Depending on the embodiment, the initial and refinedpredetermined ranges (or other criteria) can differ. As one illustrativeexample, the block alignment function is a hash function that outputs at16-bit value having 65,536 possible values. An intermediate functionsuch as a modulo or other operation can be performed on the outputvalues. For example, the intermediate function may reduce the number ofvalues that are analyzed in the block alignment determination. Forexample, a modulo-100 operation is performed such that each analyzedvalue is from between 0 and 99. The first predetermined range in theexample scenario is from 45 to 55. Where the criteria is adjusted asdescribed herein, the second predetermined range is from 40 to 60. Itwill be appreciated that the number of output values, the intermediatefunction, the first range and/or the second range can vary. In otherembodiments, the first range and the second range do not overlap or onlypartially overlap. Moreover, the range may be adjusted more than once incertain embodiments to a third refined range, fourth refined range,etc., or may revert back to previous ranges. Moreover, a modulo functionis not used in some cases, or a different intermediate function is used.

FIG. 5 illustrates a block diagram of an example memory module 502,including data segment 504, after the deduplication block alignmentshave been determined. The data segment 504 may be any size, but as anexample and not to be construed as limiting, the data segment 504 is 1MB in size. In addition, as an example and not to be construed aslimiting, the deduplication block size is a fixed 128 kB. Thus, the datasegment 504 could potentially contain a total of eight deduplicationblocks. As described above, with reference to FIGS. 2-4, a blockalignment function is performed on the data within a sliding window toproduce a block alignment value. The sliding window can be the same sizeas a deduplication block size. As described above, if the blockalignment function returns a hit a deduplication block is establishedand the sliding window moves relative to the data segment according tothe size of the deduplication block. If the block alignment functiondoes not return a hit, the sliding window moves relative to the datasegment according to the size of an incremental value. As discussedpreviously, the incremental value may be any number of different sizesranging from one data subcomponent or more. A block alignment functionis performed on the data within the moved sliding window, and theprocess continues. FIG. 5 illustrates a data segment after this processhas occurred across the entire data segment. Data segment 504 containsvarious deduplication blocks (506, 508, 510, 512, 514, 516, 518) offixed size. In this example, the fixed size for each deduplication blockis 128 kB. However, as mentioned previously, the fixed size may be anysize. The data segment also contains various gaps (520, 522, 524, 526,528) interspersed between the various deduplication blocks (506, 508,510, 512, 514, 516, 518). The gaps vary in size depending on the numberof unsuccessful outputs of the block alignment function. The moreunsuccessful outputs from the block alignment function, the larger thegap.

Reviewing the data segment 504 indicates that there were no hits on thefirst bytes or other units of data within the sliding window. The firsthit occurred at block1 506. No hits occurred between block1 506 andblock2 508, resulting in gap 522. The same can be said for gaps 524,526, and 528. FIG. 5 also illustrates that the block alignment of a datasegment 504 will likely result in fewer deduplication blocks beingestablished than could potentially be established. For example, if therewas a hit each time a block alignment function was performed on the datawithin the sliding window, a total of eight would be established.However, usually some amount of data will not return a hit, and thus,there will be fewer deduplication blocks than theoretically possible,with gaps therebetween. In the example illustrated in FIG. 5, only sevendeduplication blocks of 128 kB each are established within the 1 MB datasegment. Thus, there is 128 kB of gap data.

FIGS. 6-10 are flow diagrams illustrative of various processes orroutines for defining deduplication block alignments within a datasegment. FIG. 6 is a flow diagram illustrative of one embodiment of aroutine implemented by a storage system for defining deduplication blockalignments within a data segment. FIG. 7 is a flow diagram illustrativeof one embodiment of a routine implemented by a storage system fordeduplicating data blocks within a data segment. FIG. 8 and FIG. 9 areflow diagrams illustrative of different embodiments of a routineimplemented by a storage system for defining deduplication blockalignments within a data segment. FIG. 10 is a flow diagram illustrativeof another embodiment of a routine implemented by a storage system fordefining deduplication block alignments within a data segment. The FIGS.6-10 will now be described in detail.

FIG. 6 is a flow diagram illustrative of one embodiment of a routine 600implemented by a storage system for defining deduplication blockalignments within a data segment. The storage system may be any one ofthe components discuss above with reference to FIG. 1. For instance thestorage system can be the client 185, the data agent 195, the storagemanager 102, or any one of the media agents 105. One skilled in therelevant art will appreciate that the elements outlined for routine 600may be implemented by one or many storage systems/components.Accordingly, the following illustrative embodiments should not beconstrued as limiting. In addition, it is to be understood that theorder of the blocks may be changed without affecting the nature or scopeof the description.

At block 602, the routine is initiated. At block 604 a storage systempositions a sliding window at the beginning of a data segment. However,it is to be noted that the sliding window can be placed at any locationbefore, within, or after the data segment. Thus, the initial location ofthe sliding window before, within, or after the data segment may also bereferred to as the beginning of the data segment. As mentioned above,the data segment can be a stream of data or can be data that is storedin computer memory. The computer memory can be any of RAM, ROM,solid-state, tapes, hard-discs, magnetic storage disks, or the like. Thestorage system can select the beginning of the data segment based onpredetermined criteria or may select the beginning dynamically.

Once the sliding window has been positioned at the beginning of the datasegment to be analyzed, the storage system performs a block alignmentfunction on the data within the sliding window, as illustrated at block606. As described above, the block alignment function may be any numberof functions that allows the storage system to determine an appropriatealignment. Hash functions may provide one alternative for the blockalignment function. The hash function may be either a weak or stronghash function. The type of hash function used can be determined based onthe amount of computing resources available, time sensitive nature ofthe deduplication routine, and the like. As is known in the art, weakhash functions typically require less resources than strong hashfunctions. Thus, in determining the boundary alignment, a weak hashfunction may be preferred.

With continued reference to FIG. 6, at decision block 610, the storagesystem determines if the output of the block alignment functionsatisfies a predetermined criteria. In one embodiment, the predeterminedcriteria can be a range of values. For instance, if a hash function isused, a range of values from the potential output values of the hashfunction may be selected. After the hash function, or other blockalignment function, has been performed on the data within the slidingwindow, if the output of the block alignment function is a value withinthe range selected, the storage system may determine that thepredetermined criteria is met. The range of values may vary depending onthe data segment being analyzed and the block alignment function beingused.

If the predetermined criteria is met the storage system establishes adeduplication block, as illustrated in block 610. Establishing adeduplication block may include storing any information that would aidin locating the deduplication block. For instance, in one embodiment,the storage system can store the location of the sliding window, thesize of the deduplication block, a pointer to or address of thededuplication block, as well as the output of the block alignmentfunction. In another embodiment, the storage system can store areference to the deduplication block and the size of the deduplicationblock may be predetermined. In yet another embodiment, the storagesystem can store the entire deduplication block for later use. Anynumber of methods may be implemented that allows the storage system toaccess the data within the established deduplication block withoutdeparting from the spirit and scope of the description.

Upon establishing the deduplication block, the storage system moves thesliding window relative to the data segment based on the size of thededuplication block, as illustrated in block 612. As noted earlier, thesize of the deduplication block may be predetermined or may bedetermined dynamically. Moving the sliding window can be accomplished ina number of different ways depending on the data. In one embodiment, thesliding window is a buffer. In this embodiment, moving the slidingwindow includes changing the data within the buffer with new data to beanalyzed. In some embodiments, all of the data within the buffer isreplaced, in other embodiments some of the data remains. In anotherembodiment, the sliding window is a pointer or set of pointers to one ormore address locations within the data segment. In this embodiment,moving the sliding window can include changing the address locationpointed to by the pointer. In an embodiment where the data is streaming,moving the sliding window can be as simple as analyzing the datacurrently passing the system. In addition, based on the starting pointof the sliding window, the sliding window can move forwards or backwardsrelative to the data segment.

Furthermore, the sliding window can move in increments larger or smallerthan the size of the deduplication blocks. For example, moving thesliding window based on the size of the deduplication block may resultin gaps between the newly created deduplication block and the next datawithin the sliding window, or some data may have the block alignmentfunction performed more than once. If the predetermined criteria is notmet, then the storage system moves the sliding window relative to thedata segment by an incremental amount, as illustrated at block 614. Asmentioned above, the incremental amount may be any one of many differentamounts. For example, the incremental amount maybe a bit, a byte,several bytes, hundreds, thousands, or even more bytes depending on thedata segment.

Once the sliding window moves relative to the data segment as discussedwith reference to either block 612 or 614, the storage system determinesif there is more data in the data segment, as illustrated in block 616.If there is no more data in the data segment, the storage system can endthe process, as illustrated in block 618. As part of this determination,the storage system can determine if there is sufficient data left withwhich to make a deduplication block. If there is insufficient dataremaining in the segment to make a deduplication block, the storagesystem can determine that the end of the segment has been reached, eventhough some data remains, and end the routine, as illustrated in block618.

If, however, the storage system determines that additional data remainswithin the data segment, the storage system can continue determining theblock alignment for the additional data and perform the block alignmentfunction on the data within the moved sliding window, as illustrated inblock 606. In addition, as part of the end of segment determination indetermining block 616 if storage system determines that there issufficient data remaining in the data block to establish only one moredata block without any intermediate gaps or gaps at the end of the datablock, the storage system can establish a deduplication block withoutperforming a block alignment function.

With further reference to FIG. 6, it is to be understood that the orderof the blocks may be changed without affecting the nature or scope ofthe description, or the routine 600 may be performed with more blocks orfewer blocks. For example, although not illustrated, routine 600 maystore the output of the block alignment function as well as the locationof the alignment window.

FIG. 7 is a flow diagram illustrative of one embodiment of a routine 700implemented by a storage system for deduplicating data blocks within adata segment. One skilled in the relevant art will appreciate that theelements outlined for routine 700 may be implemented by one or manystorage systems/components. Accordingly, the following illustrativeembodiments should not be construed as limiting.

At block 702, the routine is initiated. At block 704, the storage systemperforms a block comparison function on a first data block. The firstdata block may be the first established data block from block 610 ofFIG. 6. In an embodiment, routine 700 follows routine 600. In such anembodiment, storage system determines block alignments and establisheddata blocks using routine 600, and uses routine 700 to deduplicate theblocks established by routine 600. The block comparison function may beany number of various functions. In one embodiment, the block comparisonfunction may compare each subcomponent of data in two data blocks todetermine if the current deduplication data block is equivalent toanother, previously stored deduplication data block. For example, theblock comparison function may compare the hash of the currentdeduplication data block to the hashes of previously storeddeduplication data blocks. In one embodiment the hash function used maybe different from the hash function used in routine 600. For example, inorder to reduce or eliminate collisions in which non-redundant data isremoved, the hash function can be a strong hash function. In oneembodiment, the block comparison function compares the data blockswithin the data segment to other data blocks within the data segment. Inother embodiments, the block comparison function compares data blocks inone data segment with data blocks in other data segments. Any number ofvarious functions may be used to determine if a data block is equivalentor substantially equivalent to another data block without departing fromthe spirit and scope of the description.

Once the block comparison function is performed on the data block, thestorage system may determine if the output of the block comparisonfunction meets a predetermined criteria. Generally, the predeterminedcriteria can be whether the output block comparison function indicatesthat current deduplication block matches another, previously storeddeduplication data block, such as where the output is equal to theoutput for another, previously stored deduplication data block. Thevalue of the output of the block comparison function on the other datablock can be stored in computer memory and can be from the same datasegment or from a different data segment. For example, the system in oneembodiment maintains a table of hash values corresponding to previouslystored deduplication data blocks. If the storage system determines thatthe output of the block comparison function meets the predeterminedcriteria (e.g., the hash matches that of another stored data block), thestorage system replaces the data block with a reference to the otherdata block. In this manner, the storage system can reduce the overallsize of the data segment. Rather than containing all the data from thedata block, the data segment will contain a reference to an equivalentdata block.

If the output of the block comparison function does not meet thepredetermined criteria the storage system can determine if there areadditional data blocks, as illustrated in block 710. In addition, afterthe storage system has replaced a data block with a reference, asillustrated in block 708, the storage system will then determine ifthere are additional data blocks, as illustrated in block 710. If thereare additional data blocks, the storage system can perform the blockcomparison function on the next data block, as illustrated in block 712.If, however, there are no more data blocks, then the storage system canend the routine 700, as illustrated in block 714.

With further reference to FIG. 7, it is to be understood that the orderof the blocks may be changed without affecting the nature or scope ofthe description, or the routine 700 may be performed with more blocks orfewer blocks. For example, although not illustrated routine 700 maystore the output of the block comparison function.

FIG. 8 and FIG. 9 are flow diagrams illustrative of alternativeembodiments of routines 800 and 900 implemented by a storage system fordefining deduplication block alignments within a data segment. Oneskilled in the relevant art will appreciate that the elements outlinedfor routines 800 and 900 may be implemented by one or many storagesystems/components. Accordingly, the following illustrative embodimentsshould not be construed as limiting.

FIG. 8 and FIG. 9 further illustrate alternative embodiments foraltering, or refining, the predetermined criteria during deduplication.As mentioned above, refining the criteria as the sliding window movesacross the data segment can further increase the likelihood of findingidentical data and reducing the amount of memory used by the datasegment. At a high-level, the process 800 refines the criteria and thenreevaluates at least some of the data within the data segment. This mayoccur, for instance, when refining the criteria includes changing theblock alignment function. The process 900, on the other hand, refinesthe criteria and continues evaluating the data segment withoutreevaluating previously evaluated data. This may occur, for example,when refining the criteria includes expanding, or changing, the rangeconsidered to be a hit. Generally, any of the alignment criteriaadjustment techniques described herein such as those described withrespect to any of FIGS. 4, 8 and 9 may be performed by a criteriaadjustment module executing on one or more of the components in thesystem. For example, a criteria adjustment module may be implemented onthe storage manager, one or more of the media agents, or a combinationthereof.

Blocks 802-812 and blocks 902-912 of FIG. 8 and FIG. 9, respectively,are similar to and correspond with blocks 602-612 of FIG. 6. At block802 and block 902 the storage system, positions the sliding window atthe beginning of a data segment. At blocks 804 and 904, the storagesystem performs a block alignment function on the data within thesliding window. At decision blocks 808 and 908, the storage systemdetermines if the output of the block alignment function satisfies apredetermined criteria. If the output of the block alignment functionsatisfies a predetermined criteria the storage system establishes adeduplication block, as illustrated in blocks 810 and 910, and moves thesliding window relative to the data segment based on the size of thededuplication block, as illustrated in blocks 812 and 912.

In blocks 814 and 914, the storage system resets a gap counter, whichwill be explained in greater detail below. Blocks 816-818 of FIG. 8 andblocks 916-918 of FIG. 9 correspond with block 616-618 of FIG. 6. Inblocks 816 and 916, the storage system determines if the end of thesegment has been reached. Similar to block 616 of FIG. 6, the storagesystem may make this determination in any number of different ways. Ifthe storage system determines that the end of the data segment has beenreached, the storage system ends the routine, as illustrated in blocks818 and 918.

Referring now to only FIG. 8, if the storage system determines atdetermination block 808 that the output of the block alignment functiondoes not satisfy a predetermined criteria, the storage system incrementsa gap counter, as illustrated in block 820. The gap counter tracks thenumber of consecutive instances that the data in the sliding window hasfailed to satisfy the predetermined criteria. As mentioned previously,when the output of the block alignment function satisfies apredetermined criteria the gap counter is reset. Thus, by incrementingeach time the output of the block alignment function fails to satisfythe predetermined criteria, the size of the gap between eachdeduplication block can be monitored. The gap counter may be implementedin hardware or software in any number of ways without departing from thespirit and scope of the description.

With continued reference to FIG. 8, the storage system determines if athreshold number of no-hits has been reached, as illustrated in block822. In some embodiments, if there have not been enough hits, i.e. therehave not been enough outputs from the block alignment function thatsatisfy the predetermined criteria, it may be beneficial to adjust thepredetermined criteria to increase the number of hits within a datasegment. Thus, a threshold number of gaps, or no-hits, can indicate whenthe predetermined criteria should be adjusted.

If at decision block 822, the storage system determines that the gapcounter has not reached a threshold number, the storage system moves thesliding window relative to the data segment based on an incrementalvalue, as illustrated in block 824, similar to block 614 of FIG. 6.

On the other hand, if at determination block 822, the storage systemdetermines that the threshold has been reached, the storage system mayadjust the predetermined criteria and reset the gap counter, asillustrated in blocks 826 and 828, respectively. In adjusting thepredetermined criteria, the storage system may perform any number oftasks, such as increase the range of acceptable values, establish a newrange of acceptable values different from the previous range, or canadjust the block alignment function. In one embodiment, the storagesystem may adjust the block alignment function by using a different hashfunction. In another embodiment, the storage system can adjust the sizeof the sliding window.

Upon adjusting the predetermined criteria, as illustrated in block 826,and resetting the gap counter, as illustrated in block 828, the storagesystem may then position the sliding window at the beginning of the datasegment and begin computing the block alignment function from thebeginning, as illustrated in block 804. Although not illustrated in FIG.8, in some embodiments only some of the data within the data segment isreevaluated. For instance, the storage system moves the sliding windowto an earlier location within the data segment that is not the beginningof the data segment, or the initial location of the sliding window. Analternative embodiment is also described with reference to FIG. 9, wherethe sliding window moves based on an incremental value after thecriteria is refined.

Blocks 920 and 922 of FIG. 9 correspond with blocks 820 and 822 of FIG.8. At block 920, the gap counter is incremented. At block 922, thestorage system determines if the gap counter has reached a thresholdnumber, similar to block 822 of FIG. 8. Also similar to FIG. 8, if thegap counter has not reached the threshold value, the storage systemmoves the sliding window relative to the data segment according to anincremental value, as illustrated in block 924 and makes the end ofsegment determination, as illustrated in block 916, and described ingreater detail above.

However, if the storage system determines that the gap counter hasreached a threshold value, the storage system adjusts the predeterminedcriteria, as illustrated in block 926. Adjusting the predeterminedcriteria may be done in a similar manner as described earlier withreference to block 826 of FIG. 8. The storage system can also reset thegap counter, as illustrated in block 928, and described in greaterdetail above, with reference to block 828 of FIG. 8.

With further reference to FIG. 8 and FIG. 9, it is to be understood thatthe order of the blocks may be changed without affecting the nature orscope of the description. In addition, routines 800 and 900 may beperformed with more blocks or fewer blocks. For example, although notillustrated routines 800 and 900 may store the output of the blockalignment function. In addition, routines 800 and 900 may reset the gapcounter before establishing a deduplication block, moving the slidingwindow, or adjusting the predetermined criteria. Thus, there are variousways of implementing routines 800 and 900 without departing from thespirit and scope of the description.

FIG. 10 is a flow diagram illustrative of another embodiment of aroutine 1000 implemented by a storage system for defining deduplicationblock alignments within a data segment. At block 1002, the routine isinitiated. At block 1004, the storage system determines if the gap sizeis equal to a threshold size. As discussed in greater detail above, thegap counter is able to track the number of no-hits and size of the gapbetween deduplication blocks. In some instances, the size of the gapbetween deduplication blocks may be the same as, or larger than adeduplication block. In such a scenario, it can be beneficial to createa deduplication block out of the gap. Thus, at block 1004, the storagesystem determines if the gap size is equal to or larger than adeduplication block. If the storage system determines that the gap sizeis not equal to or larger than a deduplication block, the storage systemmoves the sliding window relative to the data segment based on anincremental value, as illustrated in block 1014, and described ingreater detail above with reference to block 614 of FIG. 6.

However, if the storage system determines that the gap size is equal toor greater than a deduplication block, the storage system may establisha deduplication block out of the data in the gap, as illustrated inblock 1006. Although not illustrated in FIG. 10, the storage system mayalso establish a deduplication block out of a portion, or portions, ofthe data in the gap. The storage system can establish the deduplicationblock in a manner similar to that described above with reference toblock 610 of FIG. 6. The storage system moves the sliding windowrelative to the data segment based on the size of the newly establisheddeduplication block, as illustrated in block 1008, and described ingreater detail above with reference to block 612 of FIG. 6. Although notillustrated in FIG. 10, in some embodiments, the sliding window may notbe incremented after a block is created out of the data, or portionsthereof. For instance, if a block is created once a gap is the size of adeduplication block, the sliding window may already be positioned in theappropriate location, i.e. over the data that comes after the data inthe newly created deduplication block. In other embodiments, the slidingwindow may be incremented by an incremental value. At block 1010, thestorage system can reset the gap counter, as described in greater detailabove with reference to block 814 of FIG. 8. At block 1012, the storagesystem can stop the routine 1000.

It is to be understood that routine 1000 can be implementedindependently or as part of any of routines 600, 700, 800, and/or 900.As such, the determination of whether a deduplication block should beestablished from a gap of data can be done along with the otherdeterminations made in routines 600, 700, 800, and 900. For example,routine 1000 may be inserted in the place of block 614 of FIG. 6, or inother locations within any of routines 700, 800, or 900.

With further reference to FIG. 10, it is to be understood that the orderof the blocks may be changed without affecting the nature or scope ofthe description, or the routine 1000 may be performed with more blocksor fewer blocks. In addition, the blocks of routine 1000 may beperformed in parallel or interleaved with the blocks of routines 600,700, 800, or 900. Thus, there are many ways of implementing routine 1000without departing from the spirit and scope of the description.

It will be appreciated by those skilled in the art and others that allof the functions described in this disclosure may be embodied insoftware executed by one or more processors of the disclosed componentsand mobile communication devices. The software may be persistentlystored in any type of non-volatile storage.

Conditional language, such as, among others, “can,” “could,” “might,” or“may,” unless specifically stated otherwise, or otherwise understoodwithin the context as used, is generally intended to convey that certainembodiments include, while other embodiments do not include, certainfeatures, elements and/or steps. Thus, such conditional language is notgenerally intended to imply that features, elements and/or steps are inany way required for one or more embodiments or that one or moreembodiments necessarily include logic for deciding, with or without userinput or prompting, whether these features, elements and/or steps areincluded or are to be performed in any particular embodiment.

In certain embodiments of the invention, operations disclosed herein canbe used to copy or otherwise retrieve data of one or more applicationsresiding on and/or being executed by a computing device. For instance,the applications may comprise software applications that interact with auser to process data and may include, for example, database applications(e.g., SQL applications), word processors, spreadsheets, financialapplications, management applications, e-commerce applications,browsers, combinations of the same or the like. For example, in certainembodiments, the applications may comprise one or more of the following:MICROSOFT EXCHANGE, MICROSOFT SHAREPOINT, MICROSOFT SQL SERVER, ORACLE,MICROSOFT WORD and LOTUS NOTES.

Moreover, in certain embodiments of the invention, data backup systemsand methods may be used in a modular storage management system,embodiments of which are described in more detail in U.S. Pat. No.7,035,880, issued Apr. 5, 2006, and U.S. Pat. No. 6,542,972, issued Jan.30, 2001, each of which is hereby incorporated herein by reference inits entirety. For example, the disclosed backup systems may be part ofone or more storage operation cells that includes combinations ofhardware and software components directed to performing storageoperations on electronic data. Exemplary storage operation cells usablewith embodiments of the invention include CommCells as embodied in theQNet storage management system and the QiNetix storage management systemby CommVault Systems, Inc., and as further described in U.S. Pat. No.7,454,569, issued Nov. 18, 2008, which is hereby incorporated herein byreference in its entirety.

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser, or via other means suitable for thepurposes described herein. Data structures described herein may comprisecomputer files, variables, programming arrays, programming structures,or any electronic information storage schemes or methods, or anycombinations thereof, suitable for the purposes described herein. Userinterface elements described herein may comprise elements from graphicaluser interfaces, command line interfaces, and other interfaces suitablefor the purposes described herein.

Embodiments of the invention are also described above with reference toflow chart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products. It will be understood that eachblock of the flow chart illustrations and/or block diagrams, andcombinations of blocks in the flow chart illustrations and/or blockdiagrams, may be implemented by computer program instructions. Thesecomputer program instructions may be provided to a processor of ageneral purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions, which execute via the processor of the computer orother programmable data processing apparatus, create means forimplementing the acts specified in the flow chart and/or block diagramblock or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to operate in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the acts specified in the flow chart and/or blockdiagram block or blocks. The computer program instructions may also beloaded onto a computer or other programmable data processing apparatusto cause a series of operations to be performed on the computer or otherprogrammable apparatus to produce a computer implemented process suchthat the instructions which execute on the computer or otherprogrammable apparatus provide steps for implementing the acts specifiedin the flow chart and/or block diagram block or blocks.

While certain embodiments of the inventions have been described, theseembodiments have been presented by way of example only, and are notintended to limit the scope of the disclosure. Indeed, the novel methodsand systems described herein may be embodied in a variety of otherforms; furthermore, various omissions, substitutions and changes in theform of the methods and systems described herein may be made withoutdeparting from the spirit of the disclosure. The accompanying claims andtheir equivalents are intended to cover such forms or modifications aswould fall within the scope and spirit of the disclosure.

What is claimed is:
 1. A method for defining deduplication blockalignments within a data segment, the method comprising: iterativelyperforming a block alignment function on data within a sliding window ina data segment, and following each iterative performance of the blockalignment function: in response to determining that the output of theblock alignment function performed on a current window of data satisfiesa predetermined criteria: establishing with one or more computerprocessors a deduplication data block having a predetermined block size;and moving the sliding window relative to the data segment by an amountbased on the predetermined block size before a next iterativeperformance of the block alignment function; and in response todetermining that the output of the block alignment function performed onthe current window of data does not satisfy the predetermined criteria:moving the sliding window relative to the data segment by apredetermined incremental amount that is different from thepredetermined block size before the next iterative performance of theblock alignment function and without establishing a deduplication datablock, wherein gaps of data not belonging to any deduplication datablock exist between at least some established deduplication data blocksfollowing performance of the block alignment function across the datasegment, and wherein the size of the gaps of data is based at least on anumber of successive outputs of the block alignment function that do notsatisfy the predetermined criteria.
 2. The method of claim 1, whereineach of the established deduplication data blocks have the samepredetermined block size.
 3. The method of claim 1, wherein the gaps ofdata comprise variable gap sizes.
 4. The method of claim 1, furthercomprising changing the predetermined criteria in response todetermining, for a threshold number of consecutive iterations, that theoutput of the block alignment performed on the data within the slidingwindow does not satisfy the predetermined criteria.
 5. The method ofclaim 4, wherein the predetermined criteria is whether the output of theblock alignment function falls within a predetermined range, andchanging the predetermined criteria comprises altering the predeterminedrange.
 6. The method of claim 4, wherein the changing the predeterminedcriteria comprises altering the block alignment function.
 7. The methodof claim 6, wherein the block alignment function is a hash function andaltering the block alignment function comprises using a different hashfunction.
 8. The method of claim 4, further comprising iterativelyperforming the block alignment function across at least a portion of thedata segment again in response to changing the predetermined criteria.9. The method of claim 1, further comprising: in response to determiningfor a threshold number of consecutive iterations that the output of theblock alignment function performed on the current window of data doesnot satisfy the predetermined criteria: establishing a deduplicationdata block; and moving the sliding window relative to the data segmentby an incremental amount.
 10. The method of claim 1, wherein a size ofthe gaps of data correspond to a size of one or more incrementalamounts.
 11. A deduplication system configured to define deduplicationblock alignments within a data segment, the system comprising: adeduplication block alignment module executing in one or more processorsand configured to iteratively perform a deduplication block alignmentfunction on data within a sliding window in a data segment and which,for each iterative performance of the block alignment function, isconfigured to: establish a deduplication block having a predeterminedblock size in response to determining that the output of thededuplication block alignment function performed on the data within thesliding window satisfies a predetermined criteria; and define at least aportion of data having a predetermined incremental size that isdifferent from the predetermined block size as not belonging to adeduplication block in response to determining that the output of theblock alignment function performed on the data within the sliding windowdoes not satisfy the predetermined criteria, wherein the predeterminedincremental size is determined prior to determining that the output ofthe block alignment function performed on the data within the slidingwindow does not satisy the predetermined criteria.
 12. The system ofclaim 11, wherein the block alignment module is further configured to:move the sliding window relative to the data segment by an amount basedon the predetermined block size in response to determining that theoutput of the block alignment function performed on the data within thesliding window satisfies the predetermined criteria; and move thesliding window relative to the data segment by an incremental amount inresponse to determining that the output of the block alignment functionperformed on the data within the sliding window does not satisfy thepredetermined criteria.
 13. The system of claim 12, wherein the blockalignment module is further configured to change the predeterminedcriteria in response to determining, for a threshold number ofconsecutive iterations, that the output of the block alignment functionperformed on the data within the sliding window does not satisfy thepredetermined criteria.
 14. The system of claim 13, wherein the blockalignment module is further configured to perform the block alignmentfunction across at least a portion of the data segment again in responseto the criteria adjustment module changing the predetermined criteria.15. The method of claim 13, wherein the changing the predeterminedcriteria comprises altering the block alignment function.
 16. The systemof claim 11, wherein the block alignment module is further configuredto: establish a deduplication data block in response to the blockalignment module determining for a threshold number of consecutiveiterations that the output of the block alignment function performed onthe current window of data does not satisfy the predetermined criteria;and move the sliding window relative to the data segment by anincremental amount in response to the block alignment module determiningfor a threshold number of consecutive iterations that the output of theblock alignment function performed on the current window of data doesnot satisfy the predetermined criteria.
 17. A method for definingdeduplication block alignments within a data segment, the methodcomprising: iteratively performing a deduplication block alignmentfunction on data within a sliding window in a data segment and, for eachiterative performance of the deduplication block alignment function:establishing with one or more computer processors a deduplication datablock having a predetermined block size in response to determining thatthe output of the deduplication block alignment function performed onthe data within the sliding window satisfies a predetermined criteria;and defining at least a portion of a gap of data having a predeterminedincremental size that is different from the predetermined block size asnot belonging to a deduplication data block in response to determiningthat the output of the deduplication block alignment function performedon the data within the sliding window does not satisfy the predeterminedcriteria, wherein the predetermined incremental size is determined priorto determining that the output of the block alignment function performedon the data within the sliding window does not satisfy the predeterminedcriteria.
 18. The method of claim 17, further comprising: moving thesliding window relative to the data segment by an amount that is basedon the predetermined block size before performing the next iteration inresponse to determining that the output of the block alignment functionperformed on the data within the sliding window satisfies thepredetermined criteria; and moving the sliding window relative to thedata segment by an incremental amount in response to determining thatthe output of the block alignment function performed on the data withinthe sliding window does not satisfy the predetermined criteria.
 19. Themethod of claim 18, further comprising iteratively performing the blockalignment function across at least a portion of the data segment againin response to changing the predetermined criteria.
 20. The method ofclaim 17, further comprising changing the predetermined criteria inresponse to determining, for a threshold number of consecutiveiterations, that the output of the block alignment function performed onthe data within the sliding window does not satisfy the predeterminedcriteria.
 21. The method of claim 17, further comprising: in response todetermining for a threshold number of consecutive iterations that theoutput of the block alignment function performed on the current windowof data does not satisfy the predetermined criteria: establishing adeduplication data block; and moving the sliding window relative to thedata segment by an incremental amount.
 22. A deduplication systemconfigured to define deduplication block alignments within a datasegment, the system comprising: a deduplication block alignment moduleexecuting in one or more processors and configured to iterativelyperform a deduplication block alignment function on data within asliding window in a data segment and which, for each iterativeperformance of the block alignment function, is configured to: inresponse to a determination that the output of the block alignmentfunction performed on a current window of data satisfies a predeterminedcriteria: establish a deduplication block having a predetermined blocksize, and move the sliding window relative to the data segment by anamount based on the predetermined block size before a next iterativeperformance of the block alignment function; and in response to adetermination that the output of the block alignment function performedon the current window of data does not satisfy the predeterminedcriteria: move the sliding window relative to the data segment by apredetermined incremental amount that is different from thepredetermined block size before the next iterative performance of theblock alignment function and without establishing a deduplication datablock, wherein gaps of data not belonging to any deduplication datablock exist between at least some established deduplication data blocksfollowing performance of the block alignment function across the datasegment, and wherein the size of the gaps of data is based at least on anumber of successive outputs of the block alignment function that do notsatisfy the predetermined criteria.